Skip to content

adapter: remove ENABLE_CLUSTER_CONTROLLER and the legacy REFRESH scheduler - #38101

Merged
aljoscha merged 1 commit into
mainfrom
aljoscha/cluster-legacy-01-scheduler
Aug 10, 2026
Merged

adapter: remove ENABLE_CLUSTER_CONTROLLER and the legacy REFRESH scheduler#38101
aljoscha merged 1 commit into
mainfrom
aljoscha/cluster-legacy-01-scheduler

Conversation

@aljoscha

@aljoscha aljoscha commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Part 1 of 3 of the design for removing the legacy cluster paths.
Stacked on #38078 (the design doc). Parts 2 and 3 stack on this one.

Why

The cluster controller has been default-on since v26.29 and owns the replica
set of every user managed cluster in production. It still lives behind the
break-glass dyncfg ENABLE_CLUSTER_CONTROLLER, and that flag is the only thing
keeping the legacy REFRESH scheduler reachable: both of its entry points return
before doing any work while the gate is on, so it is a strict no-op in
production today.

Removing the gate means reverting to the legacy paths requires a binary
rollback. The gate has several releases of burn-in behind it, and the direct
reshape path (see part 3) remains as the operational escape hatch.

What lands

  • The ENABLE_CLUSTER_CONTROLLER dyncfg: definition, registration, its five
    read sites, and the sqllogictest binary's force-on entry.
  • src/adapter/src/coord/cluster_scheduling.rs wholesale:
    check_scheduling_policies, check_refresh_policy,
    handle_scheduling_decisions, and the SchedulingDecision /
    RefreshDecision types.
  • Its coordinator plumbing: Message::CheckSchedulingPolicies and
    Message::SchedulingDecisions with their handlers, the timer, the
    select-loop tick, and the cluster_scheduling_decisions state.
  • The metrics mz_check_scheduling_policies_seconds and
    mz_handle_scheduling_decisions_seconds (and their rows in the generated
    doc/user/data/metrics.yml).
  • The system var cluster_check_scheduling_policies_interval, whose sole
    consumer was the timer.
  • ReplicaCreateDropReason::ClusterScheduling, constructed only by the
    scheduler.

The persisted audit vocabulary stays: SchedulingDecisionsWithReasonsV2 and
friends are written by the controller's on-refresh path too
(refresh_window_decision_to_audit_log), and old events must remain decodable
regardless.

Tests

  • test/sqllogictest/mz_cluster_schedules.slt: the
    cluster_check_scheduling_policies_interval validation block goes with the
    var.
  • test/testdrive/cluster-controller.td: the cc_handoff* legacy-handoff
    scenarios and the cc_strand break-glass section are deleted (both are
    gate-off scenarios). The freeze-the-controller technique in the
    unmanaged-conversion refusal block is replaced by cranking
    cluster_controller_tick_interval up. Part 3 adds a replacement for
    cc_strand that exercises the direct cut-over escape hatch instead.
  • test/pg-cdc/cluster-graceful-reconfiguration.td: the explicit legacy
    foreground section goes, the controller section stays.
  • misc/python/materialize/mzcompose/__init__.py pins the flag only for
    versions below v26.38, so mixed-version runs against older binaries still
    exercise the same path. parallel_workload/action.py drops it from the
    do-not-flip list, and the two launchdarkly-flag-consistency allowlist entries
    go.

…duler

The cluster controller has been default-on since v26.29 and owns the replica
set of every user managed cluster. The break-glass dyncfg kept two legacy
paths reachable: the REFRESH scheduler in `cluster_scheduling.rs`, whose two
entry points already returned before doing any work while the gate was on,
and the legacy branches in the ALTER sequencer.

Delete the gate and everything only it kept alive: the scheduler module, its
coordinator plumbing (two messages, the timer, the select-loop tick, the
`cluster_scheduling_decisions` state), the two scheduler metrics, the
`cluster_check_scheduling_policies_interval` system var, and the
`ReplicaCreateDropReason::ClusterScheduling` variant.

The persisted audit vocabulary stays: `SchedulingDecisionsWithReasonsV2` and
friends are written by the controller's on-refresh path too, and old events
must remain decodable.
@aljoscha
aljoscha force-pushed the aljoscha/cluster-legacy-01-scheduler branch from f283eea to ed282e5 Compare August 7, 2026 08:01
@aljoscha
aljoscha marked this pull request as ready for review August 7, 2026 08:14
@aljoscha
aljoscha requested review from a team and ggevay as code owners August 7, 2026 08:14
# period. A zero value would therefore crash-loop environmentd on every boot,
# so it must be rejected at ALTER SYSTEM SET time.
simple conn=mz_system,user=mz_system
ALTER SYSTEM SET cluster_check_scheduling_policies_interval = '0s';

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm, does our new controller actually run on a similar interval to where we had this before? Now I'm curious.

@aljoscha

aljoscha commented Aug 7, 2026

Copy link
Copy Markdown
Contributor Author

Close, but slower, and the difference is absorbed by a knob that was always the real one.

The old poll was cluster_check_scheduling_policies_interval, default 3s. The controller's is cluster_controller_tick_interval, default 5s. So worst-case detection latency for a scheduling decision goes 3s -> 5s.

The controller's interval is only a fallback cadence, reconcile_now cuts the sleep short after a catalog transaction changes durable cluster state. That doesn't help here though: an ON REFRESH window opening is driven by wall clock and frontier movement, not by a catalog write, so refresh scheduling really does ride the 5s tick.

Why the 2s doesn't worry me: the window decision is

write_frontier < read_ts + hydration_time_estimate

so the turn-on already leads the refresh by hydration_time_estimate. The tick only has to be small relative to that lead, and REFRESH EVERY periods are minutes to days. The one case where the 2s is visible is hydration_time_estimate unset, which defaults to 0: the window then opens exactly when the refresh is due and the cluster comes up up to 5s late instead of up to 3s late, delaying that refresh by the same amount. No correctness consequence, and if that latency matters you want a non-zero estimate anyway, which is what the option is for.

Worth saying explicitly: both numbers are the polling interval, not a bound on how long a refresh takes. Turning the cluster on is the cheap part.

Happy to drop cluster_controller_tick_interval to 3s if you'd rather keep the cadence identical. I left it at 5s because it's the controller's own tuned default and it now covers reconfiguration and autoscaling too, not just scheduling, so it's a broader tradeoff than the old scheduling-only poll.

@mtabebe
mtabebe self-requested a review August 7, 2026 13:14

@mtabebe mtabebe left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change looks good for me. Thanks for updating all the comments too

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🥳

@aljoscha

aljoscha commented Aug 8, 2026

Copy link
Copy Markdown
Contributor Author

tyty! 🙇‍♂️

you probably saw, and I don't want to hurry you, but there are some more cleanups left in this stack 😅

@aljoscha
aljoscha merged commit 38a95ce into main Aug 10, 2026
91 checks passed
@aljoscha
aljoscha deleted the aljoscha/cluster-legacy-01-scheduler branch August 10, 2026 17:52
martykulma pushed a commit to martykulma/materialize that referenced this pull request Aug 10, 2026
…nc#38102)

Part 2 of 3 of the [design][design] for removing the legacy cluster
paths.
Stacked on MaterializeInc#38101. Part 3 stacks on this one.

[design]:
https://github.com/MaterializeInc/materialize/blob/aljoscha/cluster-legacy-05-design/doc/developer/design/20260724_cluster_controller_legacy_removal.md

## Why

System/builtin clusters were excluded from the controller's ownership
(`ManagedClusterIds` and the sequencer's two ownership tests all
required
`is_user()`), so the sequencer had to keep a second, complete
replica-materialization implementation alive just for them.

The exception was originally load-bearing: the boot-time builtin replica
migration and the controller would have been two conflicting writers of
one
replica set. MaterializeInc#37929 removed that conflict.
`reconcile_builtin_cluster_replicas`
now converges a builtin cluster's replica set on the cluster's own
managed
config, the same config the controller derives its targets from, so the
two
converge on the same set by construction.

## What lands

Drop the three `is_user()` conjuncts. Runtime ALTERs of system clusters
now
flow through the controller like any other managed cluster: a
config-shape
change reshapes into a durable reconfiguration record, a factor change
updates
the config and the controller converges the replica set within a tick.

Ownership and the boot migration compose in both directions. The
controller
matches replicas by shape and count, never by name, so the
migration-created
`r1..rN` satisfy its baseline and a steady system cluster reconciles to
no
decisions. The migration converges by canonical name, so a boot after a
reshape
(or with a reconfiguration in flight) renames or re-creates replicas the
controller materialized under generator names. That is harmless churn:
every
replica is a cold process at boot anyway, so the cost is replica-id and
audit-log noise, and an in-flight record is durable, so the controller
picks the
reconfiguration back up on its first tick.

Boot ordering and 0dt read-only behavior are unchanged.
`reconcile_builtin_cluster_replicas` still runs at catalog open, which
is what
gets system clusters up and hydrated before the serve loop starts and
before a
0dt cutover. The controller cannot cover either window:
`Coordinator::bootstrap`
runs before the controller task is spawned, and the controller is
inactive while
read-only.

Also rewrites the comment in `validate_reconfiguration_resource_limits`.
The
early return for non-user clusters stays, since system clusters are
exempt from
`max_replicas_per_cluster` and credit accounting everywhere else, but
the reason
it gave ("a system cluster never reshapes into a record") is now false.

## Behavior changes

- `ALTER CLUSTER mz_system SET (SIZE ...)` goes from a synchronous
whole-set
recreation to a background graceful reconfiguration (default deadline
24h,
`ROLLBACK` on timeout). Strictly more capable: the graceful path did not
exist
  for system clusters before, and part 3 adds `WITH (WAIT FOR '0s')` for
  operators who cannot afford the overlap set.
- `ALTER CLUSTER mz_support SET (REPLICATION FACTOR 1)` updates the
config and
returns; the controller materializes the replica within a tick. Same as
user
  clusters today.

## Tests

- `test/sqllogictest/system-cluster.slt`: the block asserting a replica
count
right after a factor flip is reworked. slt does not retry, so it now
asserts
the declared factor (synchronous) and phrases the system-id property as
"no
replica that isn't system-id'd", which holds regardless of convergence.
- `test/testdrive/system-cluster.td` (which does retry) picks up the
convergence coverage: factor up materializes exactly one system-id
replica,
a `SIZE` reshape cuts the realized config over and leaves exactly one
replica
at the new size, and factor back to 0 retires it with no activity left
behind.
- `src/environmentd/tests/bootstrap_builtin_clusters.rs`: the three
assertions
  that immediately follow a runtime `ALTER CLUSTER` now poll
(`await_declared_and_actual`). The bootstrap and post-restart assertions
stay
  exact, since the catalog-open migration converges synchronously.
- `test/cluster/resources/resource-limits.td` already uses retrying
queries for
  its `mz_analytics` factor flips, so it survives unchanged and pins the
  system-cluster limit exemption end to end.
- Docs: the self-managed troubleshooting guide's `mz_catalog_server`
resize
walkthrough gets a note that the resize is a graceful reconfiguration
and
  `SHOW CLUSTERS` settles rather than flips.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants